Developing pyd3: a python API for D3

Introduction

This is my blog for creating a python API for D3 project;

First we need a broad structure that help handle communication between python and vega altair. A good visualisaion package should be intereactive, easy to use, and most importantly easy to integrete with other popular ecosystem.

The goal is: if I want a quick data visualisation of graph data I have something… if I want a quick data app I can use this tool to develop something quickly.

• Flask: Flask will encourage using base HTML file, what this means is that just using basic D3 strucutre is probably more usefull than this;
• Streamlit: Streamlit is popular data-intelligence app deployment plateform; Many of the chart use
• Jupyter + Ipython

How other opensource package integreting javascript with Python?

Vega-Altair

graph TD
    %% Python Side
    Python[Python Layer] --> |1. Chart Spec| Altair[Altair Python API]
    Altair --> |2. Vega-Lite Spec| Jupyter[Jupyter Frontend]
    
    %% JavaScript Side
    Jupyter --> |3. Render Request| VegaEmbed[vegaEmbed]
    VegaEmbed --> |4. Data Binding| VegaRuntime[Vega Runtime]
    
    %% Data Flow
    Python --> |5. Data Updates| Model[IPython Comm Model]
    Model --> |6. Signal Updates| VegaRuntime
    VegaRuntime --> |7. Visualization| DOM[DOM Rendering]
    
    %% Bidirectional Communication
    VegaRuntime --> |8. Selection Events| Model
    Model --> |9. Python Callbacks| Python

Figure 1: Python to Javascript Communication Flow implemented by package Vega-Altair

Here’s the detailed explanation of each step:

1. Python to JavaScript (Initial Setup):

   • Python creates a chart specification using Altair’s API
   • The specification is converted to Vega-Lite JSON format
   • This JSON is sent to the Jupyter frontend through IPython’s communication model

2. JavaScript Rendering:

   • The frontend uses vegaEmbed to render the visualization
   • Data is bound to the Vega runtime
   • The visualization is rendered in the DOM

3. Bidirectional Communication:

   • Python → JavaScript:
     1. Data updates are sent through the IPython comm model
     2. The model sets new values in the Vega runtime
     3. This triggers re-rendering of the visualization
   • JavaScript → Python:
     • User interactions (selections, clicks) are captured by Vega
     • These events are sent back through the comm model
     • Python receives these events and can trigger callbacks

4. Key Components:

   • IPython Comm Model: Handles the actual data transmission
   • vegaEmbed: Manages the rendering and data binding
   • Vega Runtime: Handles the actual visualization and interaction

5. Data Synchronization:

   • The system uses a debounced update mechanism (as seen in the code)
   • Changes are batched to prevent excessive updates
   • The cleanJson function ensures data is properly serialized

This architecture allows for: - Real-time updates of visualizations - Interactive features that can trigger Python code - Efficient data transfer between the two environments - Maintainable and scalable visualization code

The key to this system is the IPython comm model, which provides a reliable channel for bidirectional communication between Python and JavaScript in the Jupyter environment.

The python side API: [[https://github.com/vega/altair/blob/main/altair/vegalite/v5/api.py]]

A Heck around with IpyComm

There are python side and javascript side:

js = """
var comm_manager=Jupyter.notebook.kernel.comm_manager
var handle_msg=function(msg){
    console.log('got msg'); 
    console.log(msg)
}

comm_manager.register_target('myTarget', function(comm,msg){
    console.log('opened comm');
    console.log(msg);
    // register callback
    comm.on_msg(handle_msg)
})
"""

from comm import create_comm 
c=create_comm(target_name='myTarget',data={})
c.send('hello')

Experiment with JINJA2

Every Javascript to Python package has to use HTML template at somestage: Some of Vega-Altair’s HTML template are stored here: https://github.com/vega/altair/blob/main/altair/utils/html.pyv;

Jinja let you create HTML template and use control flow and loops as discribed by realpython: https://realpython.com/primer-on-jinja-templating/

A basic jinja2 example:

import jinja2
import jinja2
environment = jinja2.Environment()
template = environment.from_string("Hello, {{ name }}!")
template.render(name="World")

'Hello, World!'

The sample in vega2:

• The developer has setup three jinja template and able to determine which template to use using spec_to_html